remotemanager.connection.computers.base module

class remotemanager.connection.computers.base.BaseComputer(**kwargs)

Base computer module for HPC connection management.

Extend this class for connecting to your machine

unreduce_args() → None

Attempts to find any arguments who have had their chains “reduced”, and unreduce them back into dynamic variables

classmethod from_dict(spec: dict, **url_args)

Create a Computer class from a spec dictionary. The required values are:

• resources:

  a dict of required resources for the machine (mpi, nodes, queue, etc.)

• resource_parser:

  a function which takes a dictionary of {resource: Option}, returning a list of valid jobscript lines

You can also provide some optional arguments:

• required_or:

  list of resources, ONE of which is required. Note that this must be a _list_ of dicts, one for each or “block”

• optional_resources:

  as with resources, but these will not be stored as required values

• optional_defaults:

  provide defaults for the names given in optional_resources. When adding the optional arg, the optional_defaults will be checked to see if a default is provided

• host:

  machine hostname. Note that this _must_ be available for a job run, but if not provided within the spec, can be added later with url.host = 'hostname'

• submitter:

  override the default submitter

• python:

  override the default python

• extra:

  any extra lines that should be appended after the resource specification. Note that this includes module loads/swaps, but this can be specified on a per-job basis, rather than locking it into the Computer

The resources specification is in notebook:machine order. That is to say that the key is what will be required in the _notebook_, and the value is what is placed in the jobscript:

>>> spec = {'resources': {'mpi': 'ntasks'}, ...}
>>> url = BaseComputer.from_dict(spec)
>>> url.mpi = 12
>>> url.script()
>>> "--ntasks=12"

Parameters:

• spec (dict) – input dictionary

• url_args – any arguments to be passed directly to the created url

Returns:

Computer class as per input spec

to_dict(include_extra: bool = True, include_version: bool = True, collect_values: bool = True) → dict

Generate a spec dict from this Computer

Parameters:

• include_extra – includes the extra property if True (default True)

• include_extra – includes the current remotemanager version if True (default True)

• collect_value – Also collects the stored values of the arguments if True

Returns:

dict

classmethod from_yaml(filepath: str, **url_args)

Create a Computer from filepath.

Parameters:

• filepath – path containing yaml computer spec

• **url_args – extra args to be passed to the internal URL

Returns:

BaseComputer

to_yaml(filepath: str | IO | None = None, include_extra: bool = True, include_version: bool = True, collect_values: bool = True) → str | None

Dump a computer to yaml filepath.

Parameters:

• filepath – path containing yaml computer spec

• include_extra – includes the extra property if True (default True)

• collect_value – Also collects the stored values of the arguments if True

static download_file(file_url: str, filename: str) → None

Download file at url file_url and write the content out to filename

Parameters:

• file_url – url of file

• filename – name to write content to

classmethod from_repo(name: str, branch: str = 'main', repo: str = 'https://gitlab.com/l_sim/remotemanager-computers/', **url_args)

Attempt to access the remote-computers repo, and pull the computer with name name

Parameters:

• name (str) – computer name to target

• branch (str) – repo branch (defaults to main)

• repo (str) – repo web address (defaults to main l_sim repo)

Returns:

BaseComputer instance

generate_cell(name: str | None = None, return_string: bool = False) → None | str

Prints out copyable source which regenerates this Computer

Parameters:

• name (str, None) – Optional name for new computer. Defaults to new

• return_string (bool) – Also returns the string if True. Defaults to False

Returns:

(None, str)

property required: list

Returns a list of required arguments

property missing: list

Returns the currently missing arguments

property valid: bool

Returns True if there are no missing attributes

parser(resources) → list

Default parser for use on basic “SLURM style” machines.

Will iterate over resource objects, creating a script of the format:

{pragma} –{flag}={value}

..note::

This method can (and should) be overidden for a custom parser.

Parameters:

resources – Resources object, to be created by BaseComputer

Returns:

list of resource lines

script(**kwargs) → str

Takes job arguments and produces a valid jobscript

Parameters:

insert_stub (add to kwargs) – inserts the submission block stub if True (Used by Dataset)

Returns:

script

Return type:

(str)

apply_substitutions(script: list, empty_treatment: str = 'wipe') → str

Apply Substitution objects to the script

Parameters:

• script – Base script to operate on

• empty_treatment – Defines the default behaviour for valueless args wipe: deletes the whole line (default) ignore: skip the value, leaving the substitution object there local: deletes only the missing argument

remotemanager.connection.computers.base.unpack_parser(parser_data) → Function

Parser data can be either a string, dict or callable. Handle all and return the Function

Parameters:

parser_data – stored parser

Returns:

Function

remotemanager.connection.computers.base.legacy_unpack(computer: BaseComputer, payload: dict) → None

Handle legacy style dictionary specs

Parameters:

• computer – BaseComputer subclass to be updated

• payload – spec payload

Returns:

None